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(57) Abstract: Disclosed is a system that permits 
a variable number of disparate streams of datato be 
packaged together as content into a single distinct 
program referred to as anomnimedia package. A 
framework definition may be specified and created 
for theomnimedia package to allow a client set top 
box. to decode the information and provideaccess 
to streams comprising video, audio and metadata 
information. The frameworkdefinition may be 
employed by a broadcast system to create a 
broadcast stream Amechanism is described that 
permits a content provider to define a framework 
for delivering a package of related content The 
framework definition encapsulates information 
necessary to build, format, transmit and display 
the disparate contents treams. Data may be 
downloaded to a receiver prior to the broadcast 
of an associatedprogram. The present invention 
may be implemented on terrestrial, cable, satellite, 
VDSL and other transport systems, including 
those that support upstream communication. 
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SYSTEM AND METHOD TOR PROVIDING AN OMNIMEDIA PACKAGE 
. Ooss Reference to Related Applications; 

Tbis^tic8ti<mclakus^ 

'OmniMedia Package", filed November 27, 2000 by Steven Reynolds,- which is also 
specifically mcorp^rated herein by reference for all that it discloses and teaches. 

Background of the Invention 



a. TilaM of the Invention 

The present invention relates to enhanced multimedia television and more 
particularly to a system and method for organisation, combination, transmission and 
15 reception of media fiomarange of sources wherein the media may comprise aplqraffiy 
of video streams, audio streams, and other information, such as may be accessed via the 



25 



30 



Internet 



b. Ttesrrmlion o f «he ftackground 

. The format of television programs often conforms to NTSC, PAL, or SECAM 
standards wherein a predefined bandwidth, i.e. channel, is employed to carry a single 
television program. Additional information that maybe provided with a program has 
often been encoded into the vertical blanking interval (VBJ), such as closed captioning or 
alternate language support, for example. As television broadcast formats move to digital 
transmission, many programs continue to be presented in a manner similar to 'channel' . 
based television, comprising one video stream, a primary audio stream, and possibly an 

alternate audio stream. 

Continued expansion of the Internet and high bandwidth networks provides access 
to an increasing volume of information. Adoption of digital transmission formats allows 
media snch as audio, video, and metadata content, to be associated, combined, and 
presented to provide viewers with a richer and more diverse media experience. Methods 
such as MPEG7 provide for relating content information, but do not provide a method by 
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which content of various formats may be grouped, transmitted and displayed. Therefore a 
new method of organizing, transmitting, and presenting media from multiple sources is; 
needed. 

5 Summary of the Invention 

The present invention overcomes the disadvantages and limitations of the prior 
art by providing a system and method that allows a transmission system to organize and 
transmit a related set of media and for a display platform to organize and render related 
10 media information in a manner feat reflects the available media and the capabilities of the 
platform. A framework definition identifies a set of associated content (media) for a 
broadcast program. The present invention compares the format of the media with.a 
transmission format and converts media ofothor fotmats to feat of the transmission 
format. An omnimemx describes the content. Media content and the onmimenn are 
is combined into a broadcast stream and transmitted. 

The present invention may therefore comprise a method for producing a broadcast 
stream that contains audio content, video content, and metadata content comprising: 
creating a framework definition that identifies the audio content, the video content and 
the metadata content associated with a broadcast and attributes thereof, comparing the 
20 audio format of the audio content with an audio transmission format and converting the 
audio content to the audio transmission format if the audio format and the audio 
transmission format differ, comparing the video format of the video content with a video 
transmission format and converting the video content to the video transmission format if 
the video format, and the video transmission format-differ, comparing the metadata format 
25 of the metadata content with a metadata transmission format and converting the metadata 
content to the metadata transmission format if the metadata format and the metadata 
transmission format differ, creating a menu describing the audio content, the video 
content, and the metadata content, combining the audio content, the video content, and 
the metadata content into a broadcast stream, transmitting the menu; and transmitting the 
30 broadcast stream. 
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A framework controller may utilize the framework definition to access media 
content, to process and format the media content and to control packaging and 
multiplexing of the content for broadcast 

the present invention may further comprises 
5 media and metadata streams having content into a framework for distribution of the 

. .. content to a" viewer comprising: at least one video sour^^^ 
... audibsotm^fc^ •■ 
• framework controller Chat receives the video source, audio source, and metadata source 
and produces an oriinimedia package integrating the outputs into a framework, a - 
10 framework definition module that interfaces with the framework controller and defines 
all content to be used in the omnimedia package, a delivery module mat receives the 
omnimedia package from the framework controller and transmits the omnimedia package 
to areceiver, and a receiver that receives and distributes the content of the oirmimedia 
package to display devices and audio outputs, the receiver farther, coupled to at least one 
15 user input device that provides interactivity between the viewer and me receiver. 

... The present invention may utilize pre-loaded content that is transferred to a 
receiver prior to the broadcast of a media stream with which the preloaded content is 
associated. Pre-loaded content allows voluminous and complex content to be employed 
during abroadcast without requiring bandwidth to transfer me preloaded contents 
20 time of broadcast and without latencies thatmay bemeurred if the pre-loaded content 
were transferred at the time of broadcast 

The present invention may additionally comprise ametbod for rendering portions 
of a broadcast stream that contains audio content, video content, and metadata content, 
and a menu indicating the contents of me audio content, video content, and metadata 
25 content comprising: transferring preloaded metadata associated with me broadcast stream 
to a receiver prior to transmission of the broadcast stream, receiving the broadcast stream, 
displaying the menu wherein the menu includes an icon representing tlie preloaded 
metadata, receiving a user input, and rendering the preloaded metadata m response to the 
user input.. 

30 Advantageously, the present invention allows a viewer to select among a plurality 

of audio,yideo and metadata sources to obtain a televirion presentation tailored to tine 
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viewer's preferences, offering mcr^d viewing satisfaction. The present invention may 
be employed utilising the capabilities of emerging digital transmission formats to provide 
an enhanced viewing experience and to increase audience size, allowing increased . 
advertising revenue. 

IWriptlon of the Figures 

in. me figures, 

. Figure 1 is a high-level block diagram of the present invention. . 
Figure 2 depicts components of framewmk controller 108: . 
Figure 3 depicts a receiver mat may be employed with the present invention. 
Figure 4 is flow chart 400 of the operation of the metadata processor 214. 
Figure 5 is a flow chart 500 illustrating the operation of omnimenu generator 224. 

Detailed De« «-rifrtinn of the Invention 



15 



The present invention is directed to allowing the creation of larger, more robust ; 
productions with support for interactivity and multiple video, audio, and data streams. A 
framework definition provides organization of media for transmission, and for rendering 
of media by a display platform that may comprise a television, interactive television, set- 
20 top box, satellite receiver, personal computer or other equipment operable to receive data 
across a network or airwave and process data according to the method of the present 
invention. The framework definition allows a media stream or multiple media streams to 
be packaged together with other content into a single distinct program, hereinafter 
referred to as an omnimedia package. A content provider may employ the framework 
25 definition to specify and deliver a package of related content, encapsulating the 

information necessary to build, format, transmit and display the content. Content may 
comprise video, audio, and data hifOTmation that may be streamed or cached. A wide 
range of information types and formats may be employed as illustrated by table LThe 
present invention is not limited to any specific types of information, formats, or 
30 relationships between information. . • 
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Table 1 



JVlediaTvpe 


Format 


CBR and VBR encoded video streams 


MPEG, ]DVB, etc 


Encoded Audio Streams 


•mtD^ "KADI 


Packet-based encoded video/audio streams 


QuickTime, AVI, etc 


Program Synchronous Metadata 


ATVEF > Java, eio 


Scripted content 


HTML, Java, JavaScript, ECMXscnpt, VBScript, 

etc ...... 


Interactive Graphical User Interface 


HTML, CSS 1, TKM 0 


GrapHcal Eiemenis ' - 


Logos, stUl-fiwne images, JPEG, GIF, etc 



The present invention maybe employed with broadcast systems that utilize 
terrestrial, cable, satellite, VDSL, or other transport methods. The present invention may 

5 also be employed with systems in which content is requested via a reverse path and with 
systems that support local or client side storage of pre-delivered content. Digital format 
transmission systems are well suited to themethod of the present invention, but analog 
systems with VBI data and Internet connection may also be employed. 

- Figure 1 is sl high-level block diagram of the present invention, A plurality of 

10 audio/visual and metadata sources, which provide the audio, video, metadata and other 
data services, are combined to produce an omnimedia package. The term omrrimedia 
describes the inclusive nature of the content being delivered by the present invention, as 
contrasted with the usual audio/video packaging associated with television events. Thus, 
for a given event, the omnimedia package represents all of the synchronous and 

15 asynchronous components firom video $ource(s)102, audio source(s) 104, and metadata 
sources 106, which may comprise and may be associated with a primary broadcast. Video 
source(s) 102 may comprise standard television broadcast channels (analog or digital 
modulati on and encoding), any form of analog video (for example, Beta and VHS format 
tape>s), any form, of stored digital video (for example, Laserdisc, DVD or any server- 

20 based digital video, such as may be used by video-on-demand systems) or may be any 
form of packet-based streaming media. This may include alternate camera sources, 
supporting video, alternate advertising video, localized or regionalized video. Audio. 
source(s) 104 may comprise soundtracks that accompany one or more video soureeCs) 
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102, or may comprise an independent stream such as a radio sigmd orother audio only 
broadcast. Framework definition 1 10 defines the reiatkmships between different content 
streams. FoxVideo sources) 102-the^ 

■" source^) 104. Tm^^^^'^^^^ at ^^ m ^^ 
5 video sources) 102. There is no requirement that any given video source 102 have an 
accompanying audio source 104. Audio maybe encoded witha video source, such as 
may be practiced with packet-based streaming media. Metadata source(s) 106 may. 
contain data packaged with respect to an event or associated with * general subject and 
. maymclvuieexecutable^ 
to time synchronized, content related, or ancillary to an event 

Following are three examples fflustrating the types and relationships between, 
video source^) 102, audio source(s) 104, and metadata sources) 106, and how these 
sources may be organized and associated to provide a richer media presentation. In 
Example One, an omnimedia program is provided for the French Open, a tennis 
15 tournament featuring several simultaneous events. -The primary broadcast maybe from, 
the main for center) court with accompanying broadcasts from other venues. 

EXAMPLE ONE 



Source JD 



102.1. A 



104.1. A 



104.1.B 



102.13 



102.1 .C 



102.1.D 
102.1.E 



102.1.F. 



Source Type 

High bit rate video stream 
intended for fall screen 
video. 



Audio Track 



Audio Track 



Low bit rate video stream 
intended for scaled video 
. ■ window. . 



Low bit rate video stream 
intended for scaled video 
• • window. 



Medium bit rate video 

stream. 

Medium bit rate video, 
stream. 



Medium bit rate video 
stream. 



Source Description 



Video feed from center court, featuring players 

AandB: This is the Primary Video to be 
broadcast. This is the video mat is broadcast 
t o non^mnimedia enabled television systems . 
This is the PnpKsh audio track for 102.1.A. 



This is the French audio track fi>r 102. IjV. 
Camera focused on player A's coach and 
family members. 



Camera focused on player B's coach and 
family members. 



Alternate camera angle 1 from behind player : 

_____ a: 

Alternate camera angle 2 from behind player 
B- 



Video feed from court 2, featuring players C 
andD. 
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Source ID 


Source Type - 


.Source Description ■-. 


1 f\A 1 if* 


Audio -Trade 


This is the English audio track fo* 102.1.F. 


104.LD 


Audio Trade 


This i$ the French audio track fot 102-1. F. 


102.1.G 


Packet-based video stream 1 

\VXm1 CI I LLJ CilvlOvX oumu 

trade 


Video and audio feed from court 5; featuring a 
mixed doubled match between two itoorty-: 
. seeded pairs.. ... 


. • 106.1 A 


' HTML Document 


Interactive page that shows : staiisticS' for m&tch 
in progress; tigfrtly synchronized to 102,1 A 
. (video stream)- 


106.13 


HTML Document . 


Interactive page that shows summary of all 
matches in nroeress* loosely synchronized to . 
102.1.A. 102.13 and 102.1 .G (video, steeattis). 


1 AiT 1. 

• lUo.LL ■ 




Interactive naee that shows results, of todays 
matches and contains links to upcoming, 
matches {links navigate to 106.1 l^and ,v 
. 106.LE). 


106.1. D 


HTML Document 


Interactive page that provides detailed - 
commentary for an earlier match . - 


106.LE 


HTML Document 


Interactive page that shows statistics and 
: commentary for upcoming match 


106.LF 


JPEGImage. 1 


Logo image to be used by Interactive pages 
(106.LA-106.LE» 


Example ' 
fourth day of the 


Two shows the components of an otanimedia package relating to the. . 
Masters, a major golf tournament. 

. EXAMPLE TWO 


Source ID 


Source Type 


Source Description 


102.2.A 


" High bit rate video stream 
intended for full screen 
video. 


Video feed following, as appropriate and 
opportune; the various leaders: This, is th£. 
Primary Video to i>b broadcast 1 ' This i$ ! t£e 

video that is broadcast to noiiTo£ai^edia 
enabled television systems, *"*'* 


1043.A ' 


- . . Audio Track 


This is the audio track for 102>2*A~ 


102.2J3 


High bit rate video stream 
-intended for foil screen 
video. 


Video feed following Tiger Wood?. 


-104^3 


Audio Track. 


This is theaudio.track for 102,23. 




HTML Document 

.v ..... ** . . * * 


Interactive-page sbdtwfag Tiger Woods score 
. card for the round in progress- 


10623 


HTML Document 


Interactive page showing leader board for . 
-. tournament ' 


i06.2 4 ;c ; 


. 1 HTML Document 


Interactive page showing past tournament 
' . winners 
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Source Description • 


Source ID . _ 
106.2.D ; 

• 


Source TVpe 
:. Java Code 


Executable code that instructs consumer device, 
to show an animated view of the curtent hole 
in play on 102,2.A (loosely synchronized to 
102.2.A) . - 


Example! 
. retransinission of 


Tiree shows the components of am bmrimedia package relating tma 
the Mi G. Wells "War of the Worlds" radjo broadcast. 


5 


EXAMPLE 


THREE 


Source ID. 


Source Type 

Audio Track 

• 


Audio feed of the original broadcast This is 
the Primary Audio to be broadcast. This, is the 
video that is broadcast to noq^izinimed;a, . 
enabled audio systems such as an automobiles 
radio. - . - 


I 

104.3B 


High Quality Audio Track- 


This is an audio track that has been redubbed 
and digitally remastered with added stereo 

sound effects. _ 


102.3.A 


High bit rate video stream 
intended for full screen 
video. 


Video.feed of an interview with fit Q : Wells 
about the making of and hysteria gisrierked^by 
the original broadcast. V 


104-3:C 


Audio Track. 


This is the audio track for 1023 A* 


106.3.A 


HTML Document 


Interactive page showing program 
synchronous factoids about the driginaJ 
broadcast of "War of the World". Tigh tly 
synchronized to 104.&A, . 


. .106.3.B 


JPEG Image 


Program synchronous images showing photos, 
J artwork and press clippings related to l04' r 3:A. 
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Framework definition 110 des6ribesall content used in an^nmimediapacka^,'. 
Framework controller 1 08 employs framework definitioii llOto creat^an oiruumedi^ 
package. Framework definitibn 1 ^defines Hie primary video and audio to be usedby , 
non^innimedia aware platfoims, plus initial audio and video, for orrmunedia pac^ag^ /' 
streams: Framework definition 1 10 is also employed to generate a mam-menuyonmimehu..: 
• ; An ornnimenu is aninieractive user interface that presents; a menu of options available. 
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with anomnimediapackage. Options may comprise different video streams, differejiC 
television layout/different audio. sources, and omw.interactive data, streams. The •:. 
definition of each stream may include a start/stop time to enable framework corn*oller, : ~ 
108 to switch streams during when creating me^nmimedia package.: An omnimenu may. 
5 . be presented in a range of formats andmay be custonu^ by the system operator; For^ 
example, one embodiment of an ommmenii may scale current video.mto an upper right- . ;. 
. hand comer and display a list of options to the left.of altematevideq sources, alternate • - 
audio sources; and links to other data. Another embodiment of an omhimenu may employ 
a pre-defined TV layout comprising a % screen primary video format and several 
10 secondary thumbnail .vidcos and data indicators across the bottom of the screen- ; . 

Framework denratiou 1 10 may be created at a production facility or other site and 
may employ automated and manual methods of associating media. The framework, 
definition 1 10 may reflect limitations and constraints of source delivery systems, 
transmission systems, and display platform limitations. Su<&-limitations may include 
is available bandwidth, number of channels tuned by a di^lay platform, cache size, 

supported formats or Other parameters. A framework definition may specify that images, 
must be in GIF format and that HTML code must support version 3.23, for example. For 
each event or presentation, afiamework definition record may be produced that may . 
comprise information defining me type, size, nature, pricing, scheduling, and resource • 
20 requirements for each media offered. Table 2 lists a number of components mat may 
comprise a framework definition record. 

Table 2 



Framework Definition R ftr-rwd Components 



Version of the oTpnimedia package 



Ownimemi associated with the omoimedla package, may be created by frameworic controller. 



Parameters -used oy.framework controller 108 to allocate, congolaod priority trartsroi^^^ 



Flags/Gatcgoiy, languages, feature set, including mnMple ratings available (e.g. PG thru R)^ 



Peiartsteis empioyed to*ontrol delivery. (Tuning, URL, etc...) 



Price, If aay; An associated cost for fte omnimedia package 
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Time the omDirneHia package expires 

Tto iuiiber of ftwawwoi* detjiwtiaa stream records 

A plurality of stream tecbtds 
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A stream record is provided for each media source (audio,, video, or metadata) to be 
offered'-in 'file ornnimenu: Each framework definition stream record may compose:; . 

5 auriiqueiDr * 

* a i«imaryV/N selection indicator; 

♦ the media Type (audio/video/daWother); ■ 

• the.bandwidth required for the media; 
« ' a start time/date time; 

* the date and time the omniroedia package expires; 

• a description having a paragraph or two describing the data streiam '(by 

• way of example, an audio stream could include the person speaking/band. 
■' playing, a video stream could include the camera angle,' or location, and a 

data stream could describe the mteractivecontent and what it does. These 

• are used by the framework controller 108 when building the oranimemi; . 

• . a location (framework controller 108 would fill this out most likely) PID; - 

• flags category, languages, feature set, including multiple ratings available 
(e.g.PG.thruR); 

• price, including an associated cost for the stream as \vell as an additional 
fee beyond the cost of the program such mat to see a certain rating, version 
of a . show or to see video instead of just hearing audio may require a. 
premium payment; 
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. • a unique Group .ID within an Event that identifies* a group , of streams t&al 
: are u£ed by the framework controller 108 when building the onminreriu 
7(nbte that only one stream from the group can be used at one time); an4 

• any Dependent Streams, each having.a unique stream ID which is required 
to accessAise the stream. . In use, a stream may require another stream. • 
: ^ese.stream IDs may be used by the set top box to determine if the • 
. stream is available, Le.. an audio stream may be dependent on the yi<Jgo' 
stream so that only devices that can view/use the video stream can access, 
the audio streain. Also, tie stream IDs can be used by head end equipment 
that is-trying to optimize bandwidth and may want to separate different 

• streams on different transponders for the same event 

An example of a framework definition is shown below: 

\<30fa> 

. <Version>4<^Versioii> 
<M^u>Showme.btml</Menii> 

<Flags> 

<Language>English</Language> 

... <Rating?>PG-13<VRating> 

</FIags> 

<Control> ???????? </Control> 
<Pric*e>2-99<Pric6> 

* <Expire>02/17/00 21 :00</Expire> 
<StreamRecords>l </StreamRecor<Js> 
<StreamRecordl> 

<3D>459812</ID> 

<^rimaJ7>Tme</Primary> 

<MediaType>Video</MediaTyp6> 

<Bandwidtfc>256</Bandwidth> 

<Start>02/15/00 22:0(XStart> 

<Expiie>02/t7/00 21:0(XZExp2re> 

<Description></Deascription> 

<Locatioli>PID:200</Location> 

<Flags> 

<Language>£nglish<Language> 
, <Rating>PG-13</Ratin^>. 

; </Flags> 
<Price>2,99</Price> 
. <GroupID>002</GroupID> 
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<Dependan£>4598 10<yDependaiit> 
' </StreamRccordl> *' 
• . -". </xm> . . ... 

5 - Employing framework deimitt6n 110; the framework controller 108 is openfrle.to. , . 

format aad organtefr media components ^ 

the delivery system 1 1 2- Delivery system 1 12 may comprise a headend system and ; 
transmission apparatus such as cfmployed by cable, satellite, terrestrial, and othetf - \ 
broadcast systems. The organization and .e&^lishmeitf of the.stream(s) employs 
. 10 parameters provided in framework definition 110. The stream(s) is (are> delivered to 
receiver 1 14 that processes the stream(s) and provides output to display device(s) 1 16, 
audio outputs) 1 18, and may send and receive signals to/from user input device(s) 120. 

Figure 2 depicts components of framework controller 108. Framework controller 
108 includes framework control logic 216 that is operable to retrieve and interpret 

is framework definition 1 10, and employ the parameters thereof to control operation 
preprocessors 210, 212, and 214, pins packagers 218, 220, and 222 to format and 
encapsulate infoimation (video, audio and metadata) for multiplexer 226. Video 
preprocessor 210>nd audio preprocessor 212 access media streams or stored data as . 
specified by framework definition 1 1 0 and perform processing to prepare the media for 

20 the associated packager. Such processing may include rate adaptation, re-encoding, 

transcoding, format conversions, re-sampling, frame decimation, or other techniques and 
methods to generate a format and data rate suitable for packagers 2 1 8 ? 220, and 222 as 
. maybe specified by framework definition 110. Processing may include MPEG encoding 
of analog video as may be supported by encoder equipment from Divicom Inc. of 

25 Milpitas,. CA., which is a wholly owned subsidiary of C-Cube Microsystems, lac* When 
such processing is specified, framework controller 108 may provide a sequence of 
instructions to the encoder, selecting channels and controlling encoding. 

Metadata preprocessor 214 accesses metadata elements, specified by framework 
definition 110, and performs processing to prepare these the metadata packager 222v 

30 - Such processing may include transcoding^ format conversions, script conversions, script 
generation, andimage format conversions, for example. In operation* graphical metadata 
maybe sent to metadata preprocessor 214 m a computer graphics format (PhotoShop, for 
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example) that then may be converted to a forinat that the display platform recognizes (pf, 

for example). 

Figure 4 is flow chart 400 of the opera^ 
402 an image file is accessed by metadata preprocessor 214. Themetadata preprocessor . 

5 . accesses metadata and places it in a predetermined fontfat as may r oe specified, by - ' 
framework definition 1 10. Metadata may comprise graphics data, sound data, HTML 
data, VideJO data; or any other type of data. Metadata preprocessqr 214 processes the data- .* * 
arid outputs fee processed data in a transmission/miiltiplexer format. As shown in figure 
4, the flow diagram allows the image file to be converted and output by the 'metadata ■ ■ 

10 processor 21 4 in real time. At step 40< the graphic, file conversion definitions are loaded ■-. 
into the metadata preprocessor 21 4 to perform the conversion of the image file. At step 
406 the image file 402 is converted into the graphic jSLk in accordkiw with the 
definitions. At step 408 the metadata preprocessor 214 outputs the converted image file 
to the metadata packager 220. A high-speed processor may perform these functions in 

1 5 real time using definitions that can be quickly downloaded from a Thigh-speed storage 
device.. Custpm designed state machines may also be employed for format conversions 
Metadata preprocessor 214 may also be employed to convert the format of HTML 
metadata. HTML metadata maybe sent to metadata processor 214 in HTML 4.0 format 

that then may be converted to HTML 3 JL format such as maybe required by a display 

... 

20 platform. The framework controller generates commands that are sent Jo metadata * . 
processor 214 that identify the metadata and specify the output format of the metadata : 

Referring again to figure 2, video packager 21 8. and audio packager 220 are 
subsystems that package video and audio assets into a format that is compatible with the 
. deUyeiy system 1 12, including for example, packetization, serialization, rate control,. 
25 ■ null-packet insertion, and other functions required to prepare the video for compliant 
. transport via MPEG, D VB-C/S/T, PacketVideo, ot other transport formats. Fox example, 
General -Instruments (now owned by Motorola) produces a QAM modulator that . 
modulates MPEG video that is encoded from video preprocessor 210 to digital video 
broadpast (DVB) format; The flamewoik controller generates commands that are sent tp 
30 . the.QAM tfoduUtor specifying the frequency (channel) and PUD (packet identifier) for 
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the video based upon 'the framework definition that was provided for the omnimedia 

. package. ' " ■ 

As also shown in figure 2; xnetadata packager 22Zis a subsystem 

- packaging on all metadata elementsthat are to be included in the omnunedia package. 
" S: Metadata pa<^agmgmay.wmprise rate control^ pa 

■ synchronization to video and/of audio sn^ams. The metadata is also prepared for 
. transport across any c 

DVB^MHP; etc.>- A. commercially available product for performing these fimcti&ns is 
the TES3 mat is provided by Noroak Corp., Kauata; Ontario. The TES5 encoder encodes 
10 metadata into a NTSC signal.with NABTS encoding. NABTS is the protocol that allows. . 
- . metadata to be sent in the VBI (vertical blanking interval) of a NTSC signal. The 
framework controller commands the TES3 encoder, as to what lines of the VBI are 
employed for transmitting metadata. 

. Referring again to figure 2, omnimenu generator 224 may be implemented as a 
15 rules-based subsystem- employing framework definition 208 to generate a user-interface 
that presents.program options and allows a viewer to select from these. options. Rules 
may be. employed to generate an omhimenu template in HTML page format The HTML 
page may.comprise a full screen image containing buttons (active icons) that may be 
selected to activate a particular media stream. Omnimenu generator 224 employs .. 
20. framework definition 208 to identify available streams and associates each stream with a 

button.- •'<— * 

Figure 5 is a flow chart 500 illustrating the operation of omnimenu generator. 224. 

At step 502, onmimenu generator 224 accesses the fcmtewbift deiMtion 110. Atstep 
' 504, PIDS (program IDs) are extracted from the framework definition. The omnimenu 
25 template, described above, is loaded into the rnmnimftuu generator 224 at. step 506 of 

figure 5. At step 508, each of the PIDS mat have been extracted from the framework 

definition are assigned to a button such that each of the buttons is labeled with the Pip 

name. At step 5 10, selection functions are assigned to each of the buttons in accedence. 

with the labels that have been assigned to those buttons. In this manner, the video can t>e 
30 changed in accordance with/the labeled functions of each of the.buttons of the template. 

At step.512 tie template may be assigned a company logo- la this manner, the content 
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may bfe pr^erly biifaded to cpTrespond to the sourfie of that content . At step 514 me. 
omrnmemgenerator 224 saves or exports the fi cw om^mu. The onmimenu may ^>e ; 
exported to the package multiplexer 226, 
. , As shown to figure 2, package multiplexer 226 combines me package elements . , 
s " into a stream or set of streams, in preparation for transnaission. The streams are coupled 
' ' ihrbugh deUvery system mterface(s).228, to delivery system 1 12, and 1he physical and 
. logical intercohnects to the transmission system. A Divicpm, Inc., (IVfilpitas, CA) MUX 
. is^an example of apac^ 

; As indicated ro figure .1,. delivery system 1 12 then transports the streams to the 

0 ! " receiver 114. Delivery system 112 may be analog or digital. In general, system delivery 
of content is as open as needed for any particular package and topology. A simple movie 
with no. added content .may be delivered on a single broadcast multiplex, while a package 
for the Super £owl would contain many different delivery mechanisms. Some may be 
available on a particular receiver and not on others. 
15 The onmimenumayprovide adirectoryof avaaablemediaforabroadcast.event. 

The omnimenu is transmitted to a plurality of receivers 1 14. Receivers 114 may vary in 
capability, and may include upstream (reverse path) communication from the receiver to 
the headend system, or may use other return systems such as an Internet connection, for 
example. Receivers 1 14 that do not include upstream communications may employ the 
20 orrmimenu to select audio, video and metadata information contained in a broadcast 
stream. The bandwidth of an analogNTSC channel maybe employed to carry several 
; . digital video streams, audio streams, and metad^a The oinnimenu includes tuning or 
' packet mformation to identify streams mat may be accessed by the receiver. The receiver 
. ;,- includes a software program operable to display the omnimenu and operable to tune, and 

25- render, selected streams. 

In another embodiment of the present invention, receiver 1 14 supports upstream 
(»nirnnnications. The .headend system, in response to upstream communications, may 
provide on^emand'prograriuning and data delivery The omnimenu initiates upstream 
• communication in response; to user selection of a displayed media button (icon). The - . 
30 headend system may supply the requested stream in a broadcast channel, or if already 
broadcast, may provide tuning or packer decode information to the receiver. The 
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requeued stream may be employed to update the ftpmertvork definition 1 10. Framework 
controller: 10S uses th^framework definition 1 10 to allocate'bandwidth' and PID .* 
information. The framework controller provides the frequency , PID and bandwidth- 
information fiomithe framework definition and uses it to send/ooatrol video and audio 
paqkager 218/220, Video packager 218 and audio packager 220 (figure 2) then allocate 
bandwidth for each of the respective video streams. : - 

. As also shown in figure 1* infonriation from ftie framework .controller 108 is 
redded into the onmimj^ so that ^e 

streams. The frameWoric controller 108 may also include UIU/s for demand data or. . 
. strCTmuig media- Locations based oil alternate tuner systems may also be included. For 
example* a radio station-frequency having local commentary may be simulcast with the 
video. Depending of receiver 1 14 capability* some data and/or programming may be 
loaded in advance into storage built into the receiver such that the content is available 
locally for ykwingduriiig the airing of the primary content package; For example, all 
NFL player statistics may be preloaded over a trickle feed before the Super BowL In this 
manner, an interactive fantasy football application may retrieve all needed statistic 
during the game in order to let the viewer play a fantasy game during the airing of the 
primary program. 

Figure 3 depicts a receiver that may be employed in accordance with the present 
invention. Receiver 302 is used by the system end-user or viewer for use in viewing and 
controlling the omnimedia package. The receiver 302 may comprise a decoder 304, 

. parser .306, media controller 308, receiver cache 310 and receiver controller 312: Decoder 
304 may extract frameworic information from the delivery stream omnimenu. Information 

, may be. encoded in the vertical blanking interval. Decoder 304 may comprise a NABTS 
VBI decoder from ATI Technologies* Tnc, ofThorahill, Ontario. The VBI decoder may 

: extract data from the VBI and present it to framework parser 306; The data may comprise 
" an XML format file of the omftimenu. Parser 306 extracts elements that comprisetfee 
framework. Framework parser 306 receives framework data from deepder .304 and- 
prepares the data for use by the receiver controller 312. The receiver controller 3 12 may 
comprised data interpretation module. Media controller 308 selects the media streams , 
(audio, video and/or data) that are described by the framework definition. The media 
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controller comprises 308 a tuner and video/audio decoder. The media controller 308 

receives stream date from decoder 304 and control signals from receiver controller 312.- 

* .* 

Madia controller '308 selects the proper media PIDS as needed and feeds thera to display £ 
dea^s) H 6, audio dew^ 

5 • serves as the central prbcessbr for the oilier elements of receiver 302; The receiver 

♦ * Controller 312 may compare the capabilities of receiver 302 with the stream type to; - — . ' 
determine whichstoeains may be received and used by receiver 302: Receiver controller 
3 12 sends control signals to the other units in the system; The functions of receiver- 
cotitxoliei: 3 12 may be performed by software resident in the control CPU of the receive* 

10 302. Functions include receiving data from framework parser 306 and si gn al ing media 
controller 308, For example, selection of omnimenu items may alter the framework such 
that the receiver controller 312 receives information from framework parser 306 and; . . , 
signals media controller 308 to render die selected media stream. Receiver cache 3 10 
may be employed to store the framework definition and any parameters, code objects, . 

15 data Hems or other software to control omnimenu display and response to, user input. 

Receiver 302 receives data associated with a content package and employs the ^ 
data tp access the package contents. Advanced receivers may check system capabilities to 
determine which pieces of content may be rendered. For example, a digital set top box 
may be able to decode MPEG video and audio. A digital set top box may also be able to 

20 decode MP3 .audio, but not be able to decode HDTV signals. Ar^otypereceiyer inay 
only be able to decode, audio formats and, possibly, only formats not related to a Video 
signal. The receiver may also be able to decode various data types from data seprccesu , 
These may take the form of application code that would be executed on the receiver if 
. compatible. ■ The receiver of the present invention has the internal compatibility to he 

15 able to receive the omnimedia packaged signal and decode the parts that are relevant to 
its capabilities to give the fullest experience it can, regardless of die format 

, „ Referring agam to Fig- 1 , display device(s) 116 provide(s) the visual presentation 
of the framework to the end-user/vieyver, including video, streaming media, a graphical * 
\iser interface, static graphics, animated graphics, text, logos and other displayable assets. - 

«> Similarly, audio outpyt(s) 1 1 8 presents) the audio portions of the framework to the end- 

user,- including primary audio, possible secondary audio tracks, streaming media, audio 

. ♦ » • * * 

• • • 17 . " ' . • - 
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feedb;adc-froi» t^e teceiver 114, and other audio assets. User input device(s) 120^310^) . 
the user to control the receiver and interact with the framework, providing the ability to 
chtfose components of the ftamcwoik, select links within the framework, navigate the 
• graphical user interface of the reteeiyer, or perform other interactions with the receiver. . - 
: The foregoing description of the invention has been presented for purposes of . 
illustration and description. It is not intended to be exhaustive or to limit the invention icy 
the predse fbrrildisclosed, and other modific^onsand vanatfommaybeposi^lein\ 
light in the above teachings. The embodiment was chosen and described in order to; best 
explain the principles of the invention and its practical application to thereby eai^bler , \ . . 
others skilled in the art to best utilize the invention in various embodiments and various 
modifications as are suited to the particular use contemplated It is intended that the \ 
appended claims be construed to include other alternative embodiments of the invention 
except insofar as limited by the prior art 
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Claims . . • * V" * - 

Whatte claimed is: • * t .*.- _ : . . 

I. - A method for producing a broadcast stream thai contains audio content, video ;. . • • - 
; — content, and metadata content comprising: - " 

* creating a framework definition that identifies said audio content; said video. ^ 
content and said metadata content associated with a broadcast and attributes thereby. - 
5 comparing the audio format of said audio content with an audio transmission 

; format and converting said audio content to said audio transmission format if said : _ \ m / . 
\-_ . audio format and said audio transmission format differ; . < : s 

? . comparing the video format of said video content with a video transmission : 

format and converting said video content to said video transmission format if said 
10 ■ video format and said video transmission fbnfcat differ; 

" comparing the metadata format of said metadata content with a metadata: . 
* transmission format and converting said metadata content to said metadata ■" 
■ transmission format if said metadata format and said metadata transmission format 
dififerL.and . : v: 

15 .... combining said audio content, said video content, and said metadata content 

. - , into a broadcast stream*. 

.- 2. The method of claim I wherein said framework definition further comprises: . , . . 
~-y . a fiameworic definition record for each element of said audio content, said 
. video content and said metadata content- / , 

3. . The method of claim 1 wherein said menu farther comprises: 

• t • an icon for each element of said audio content, said video content, and- . 

said metadata, content 
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4. : The niethod of claim 1 wherein said metadata content comprises an image file. 

• . * 

S- The method of claim 5 wherein said converting -said metadata content further ■ . 
comprises: * " /- 

" x - ■ ';• loading said image file; 

; loading a fflc cohv^on'definition; : 
% converting said file using said conversion definition;, and 

. otrtputting ;a convdted image file. " 

6. The method of any one of the preceding claims further comprising the steps of: 

. . creating a menu describing said audio content, said video content and skid 
metadata content; 

transmitting said menu; and .*..;* v 

k * .transmitting said broadcast stream* . . 

• ~ — • 

7. A method for rendering portions of a broadcast stream that contains audio content, 
video content, and metadata content and a menu indicating the contents of said audio, 
content, video content, and metadata content comprising: 

transferring preloaded metadata associated with said broadcast stream to a 
receiver prior to transmission of said broadcast stream; 

receiving said broadcast stream; 
— displaying said menu wherein said menu includes an icon representing 

said preloaded metadata; 

receiving a user input; and 

rendering said preloaded metadata in response to said user input- 
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8. A system for combining multiple media and metadata streams having content into 
a framework for distribution of the content to a viewer, comprising: 

a framework controller that receives said video, source, audio source, and . 
' ' metadata source and produces an orrmimedia package integrating said outputs into, a 
. framework} and 

. a framework definition module that mterfaces with said framework . 
controller and defines all content to he used in tbe onrnimedia package. 

" ': ■ 

9. The system of claim 8 further comprising: 

at least one video source having an output; , . 

•«• at least ope audio source having an output; 

at least one metadata source having an output^ - i 

10. The system of claim 8 or 9 further comprising: 

a delivery module that receives said onrnimedia package from said 
framework controller and transmits said ornnimedia package.to ^receiver, and l. . ; 

'*' a receiver that receives and distributesfhe content of said orrmimedia 
- package to display devices and a)idio outputs, said receiver further coupled to atieast 
one user input device for providmg ^teractivity betweii said .viewer and me receiver. 
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Figure 2 
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Figure 3 
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Figure 4 
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Figure 5 
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